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Abstract 

In this paper we propose an approach to organizing Adaptive Problem-Based Learning (PBL) leading to the development 
of Higher-Order Thinking (HOT) skills and collaborative skills in students. Adaptability of PBL is expressed by changes 
in fixed instructor assessments caused by the dynamics of developing HOT skills needed for problem solving, flexible 
choice of control tests and problems for students, and adaptive formation of HOT skills within heterogeneous 
collaborative groups. It induces the students to develop HOT skills and collaborative skills through a combination of 
personalized and collaborative PBL. Adaptability of PBL is realized by taking into account values of the proposed 
coefficient of HOT skills development. The two-stage process of adaptive PBL allows guided development of HOT skills 
in students during study of a subject. Attention in the first stage is devoted to development of analytical HOT skills in 
students through personalized PBL. The main attention on the second stage is devoted to the development of creative 
HOT skills and collaborative skills in students through collaborative PBL. The proposed approach provides effective 
development of HOT skills and collaborative skills in students owing to: availability of a two-stage adaptive PBL process, 
complex and adaptive assessment of HOT skills, dynamic choice of control tests and problems for students, adaptive 
formation of HOT skills heterogeneous collaborative groups, and management of HOT skills development of students. 

Keywords: higher-order thinking, adaptive learning, problem-based collaborative learning 

1. Introduction 

Development of Higher-order Thinking (HOT) skills of students is a complicated multidimensional challenge of 
education. The most significant HOT skills of students are the skills needed for problem solving. The favorable 
environment for development of such skills can be created through the use of the Problem-based Learning (PBL) model 
(Duch, Groh, Allen, 2001; Amador, Miles, Peters, 2006; Bareli, 2006; Barret & Moore, 2010). Such an environment 
should be: HOT skills-centered (an instructor must focus on HOT skills development of students); student-centered (an 
instructor should encourage student interest in critical thinking and problem solving); assessment centered (an instructor 
should monitor student performance, and provide a high quantity and quality of assessment while stimulating students 
for HOT skills development); computer-supported to promote students’ HOT skills development and decrease an 
extra-heavy workload for an instructor (Hannafin & Land, 1997; Jonassen, 1998; Bransford, Brown, and Cocking, 1999; 
Baden & Major, 2004; Pettigrew & Scholter, 2010). 

There are two distinct types of HOT skills needed for problem solving: analytical and creative thinking skills. Analytical 
or logical thinking skills allow critical thinking and help select the best alternative. The analytical thinking skills are: 
ordering, comparing, contrasting, evaluating, and selecting. Creative thinking skills are also needed for problem solving. 
The creative thinking skills are problem finding (identifying the problem), efficiency (producing many ideas), flexibility 
(producing a broad range of ideas that characterizes flexible understanding), originality (producing uncommon ideas), and 
elaboration (developing ideas) (Hmelo-Silver, 2004; Bednarz, 2011; Cottrell, 2011; Cottrell, 2013). 

The goal of the proposed approach is the development of HOT skills and collaborative skills in students through 
adaptive problem-based learning. Adaptability of PBL is expressed by changes in fixed instructor assessments caused 
by dynamics of HOT skills development, flexible choice of control tests and problems for students, and adaptive 
formation of HOT skills heterogeneous collaborative groups. It should induce the students to develop HOT skills and 
collaborative skills through PBL. Adaptability of PBL is aimed at personalization of learning. 
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According to the proposed approach, development of HOT skills in students during study of a subject by the class is 
realized through a two-stage process of adaptive PBL. The course knowledge is acquired during the first stage, and is 
devoted to the development of analytical HOT skills in students. Fixed initial assessments of analytical HOT skills are 
set. After this stage, analytical HOT skills of every student are examined by the personal control questions based on 
course material. It allows assessment of development of the analytical skills. Next, adaptive collaborative group 
formation takes into account the assessment results of individual students. 

The second stage of adaptive PBL is fulfilled by the collaborative groups through a problem- solving process based on 
course material. The main attention on this stage is devoted to development of creative HOT skills and collaborative 
skills of students. After the second stage, developing HOT skills and collaborative skills of individual students are 
examined. The choice of control questions for examination of analytical HOT skills development is triggered by the 
assessments of the skills after the first stage. Development of creative HOT skills of a particular student is examined by 
control problems. 

2. Related Research 

The proposed approach is directed towards developing HOT skills and collaborative skills of students through the 
realization of adaptive PBL. The approaches, methods, and tools reviewed here are examined relative to various aspects 
of this process. 

Savery (2006) affirms that the goals of PBL are knowledge building and adoption of a problem-solving process. 
Barrows & Kelson (1995) and Hmelo-Silver (2004) determine the goals of PBL, including flexible knowledge; effective 
problem-solving skills; self-directed, lifelong learning skills; effective collaboration skills; and intrinsic motivation. 
According to Hmelo-Silver (2004), "Intrinsic motivation occurs when learners work on a task motivated by their own 
interests, challenges, or sense of satisfaction.” Students are induced to collaborate through their reflection and through 
the interdependence of learning within the group. Zimmerman (2002) specifies self-directed, lifelong learning skills 
enabling autonomous learning. 

Schmidt & Moust (2000) emphasize the particular importance of collaboration because it affects intrinsic motivation and 
learning outcomes. Mennin (2007) explores a PBL group as a complex adaptive system (CAS). CAS is characterized by 
diverse agents interacting with each other capable of changing by self-organizing. A small student group is complex. It is 
adaptive in that the participants individually and in a group are altered. Burgos, Tattersall & Koper (2006) propose 
different types of adaptive learning support including problem-solving support, information filtering, collaborative 
grouping of students, adaptive testing, and real-time course modifications by the instructor to meet the specific needs of 
learners. Brusilovsky & Peylo (2003) consider adaptive group formation using knowledge about collaborating peers, and 
adaptive collaboration support providing an interactive support of a collaboration process assisting an individual student 
in solving a problem. 

Knight (2000) pointed to the need for a systematic approach in the assessment of students’ learning that provides 
reliability, validity, and usability. Moallem (2007) considers the process of evaluation relative to the following stages; 
initial assessment, progress assessment, and product assessment. Lovie-Kitchin (2001) rightly claims assessment 
methods should be congruent with the PBL process. Macdonald and Savin-Baden (2004) stressed the need for specific 
assessment methods for PBL. The authors note assessment should reflect the practical orientation of PBL. Brookhart 
(2010) asserts that holding students accountable for HOT through assessment increases student motivation and 
improves the results of learning. Student assessment is realized through reflection on built knowledge. 

Papanastasiou (2014) defined adaptive assessment as a type of assessment that is set for each student taking into 
account an assessment of his (her) previous performance. This allows an assessment to be more accurate in terms of 
individual ability. Tilchin & Raiyn (2015) introduce an innovative approach to complex, adaptive, and 
computer-mediated assessment of HOT skills development of students. The complexity of assessment is expressed by 
forming the combined assessments of HOT skills of different types. Adaptation of assessment to the process of HOT 
skills development is expressed by changes in an instructor's fixed assessments by crossing from one phase of PBL to 
another. Assessment adaptation is provided for individual students as for a study group. Computer mediation of 
assessment is provided by a Computer Assessment Tool which promotes students’ HOT skills development and 
facilitates the assessment process for an instructor. 

The analysis of publications above shows that no approach exists for organization of adaptive PBL directed towards 
development of HOT skills and collaborative skills of students. The proposed approach will promote productive HOT 
skills development of students due to; complex and adaptive assessment, dynamic choice of control tests and problems 
for students, adaptive formation of HOT skills heterogeneous collaborative groups, and management of HOT skills 
development. 
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3. Organizing Adaptive Problem-Based Learning for Thinking Development 

Organizing adaptive PBL leading to development of HOT skills and collaborative skills of students is realized by the 

following steps: 

A. Preparation of the needed instructional materials for a course 

B. Assessment of developing analytical skills of the students after the first stage of PBL 

C. Building HOT skills heterogeneous collaborative groups of students 

D. Formation of fixed assessments of HOT skills for the second stage of PBL 

E. Assessment of HOT skills development of students after the second stage of PBL 

F. Assessment of collaborative skills of students 

A. Preparation of the needed instructional materials for a course 

Preparation of the needed instructional materials for a course includes: 

• The forming of a list of course topics 

• Formulation of course requirements from a student with regard to development of HOT skills. After studying a 
course, a student should have developed HOT skills and collaborative skills. Thereby, he (she) should be able to 
apply effectively acquired knowledge through taking creative solutions during solving of the proposed instructional 
problems, and explain cause-and-effect relationships. 

• The forming of a list instructional problems covering all learning outcomes and HOT skills 

• Setting fixed initial assessments of analytical HOT skills. The difference of the fixed initial assessments 
of the skills expresses the preference of an instructor in the development of certain skills during the first phase of 
Adaptive PBL. An example of setting the fixed initial assessments is represented by Table 1. The ‘■‘Evaluating" skill 
has the greatest need for development since it provides critical thinking. Consequently, the maximum fixed initial 
assessment is set for this skill. 

• The forming of control questions and instructional problems for the examination of HOT skills of students. The 
instructor composes: a list of control questions which should examine analytical HOT skills of students after the 
first phase of Adaptive PBL; a list of control questions and problems for examination of analytical and creative 
HOT skills of students after the second phase. 

Table 1. Fixed assessments of analytical HOT skills 


The type of thinking skills 

Skills name 

The fixed initial assessments (%) 


Evaluating 

14 


Selecting 

10 

Analytical (40%) 

Contrasting 

7 


Comparing 

5 


Ordering 

4 


B. Assessment of developing analytical skills of students after the first stage of PBL 

A student’s analytical HOT skills are assessed (by an instructor) through the answers given by a student to control 
questions from the individual list. An instructor assessment of a student answer to a question varies from zero to the 
fixed initial assessment of an analytical skill corresponding to a question. If a student cannot answer a question, or an 
answer is incorrect, then an instructor assessment is equal to zero. If a student answers a question corresponding to a 
certain skill correctly, an instructor assessment is equal to a fixed initial assessment of the corresponding skill. It means 
he or she has developed this skill. If a student answer is not full, an instructor sets a suitable assessment. As a result, an 
assessment of a student is made. Assessments of other students from a study group are set analogously. An example of 
assessment of the analytical skills of students is represented by Table 2. Formative assessments of analytical skills in 
student s h s 3 , and s 5 are 34, 19, and 14, accordingly. 

Table 2. Assessment of analytical skills after the first stage of PBL 


The questions 

Skills names 


SI 

S2 

S3 

S4 

S5 

S6 

q a n 

Evaluating 

7% 

7 

2 

3 

5 

2 

4 

q a i 2 

Evaluating 

7% 

7 

2 

2 

4 

0 

3 

q a 2 i 

Selecting 

5% 

4 

3 

5 

3 

0 

5 

q a 22 

Selecting 

5% 

4 

2 

5 

4 

0 

5 

q a 3i 

Contrasting 

7% 

7 

4 

2 

7 

3 

4 

q a 4i 

Comparing 

5% 

3 

5 

2 

4 

5 

2 

q a si 

Ordering 

4% 

2 

0 

0 

0 

4 

0 


Formative assessment of analytical skills 

34 

18 

19 

27 

14 

23 


The obtained assessments allow an instructor to assess developing analytical skills of students through comparison of 
received assessments with fixed initial assessments by using a formula: 
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5(k 1 ) = (g f (k i )-g(k i ))/g(k i ), -1< 5(k ( ) < 0 (1) 

where 

5(k ; ) is a coefficient of development of skill k, in students of the study group; 

g f (kj) is a total formative assessment of skill k; for the study group. It is the equal sum of formative assessments of skill 
kj of study group students; 

g (kj) is a total fixed assessment of skill k,. It is determined by multiplication of the fixed assessment of the skill kj on 
the number of students in the study group. 

Example 1: 

There are six students Si, S 2 ,..., S6 in the study group. The formative assessments of the “Evaluating” skill for these 
students are 14, 4, 5, 9, 2, 7, accordingly (Table 2). Hence, a total formative assessment of this skill for the study group 
is equal to 41. Fixed assessment of this skill is equal tol4 (Table 1). Consequently, a total fixed assessment of the skill 
for the study group comprising six students equals 84. Then, the value of the coefficient of development of the 
“Evaluating” skill in the study group, according to formula (1) is 5 (kj) = - 0.51. Analogously, the values of the 
coefficient of development of “Selecting”, “Contrasting”, “Comparing”, and “Ordering” skills are - 0.33, - 0.36, - 0.3, 
and - 0.75, accordingly. 

C. Building HOT skill heterogeneous collaborative groups of students 

Effective PBL is created by the following conditions for building a collaborative group: 

• All the students studying a course should have all analytical HOT skills. Students develop needed HOT skills as 
a result of collaboration with other students in a study group. 

• Maximum mutual supplementation of skills of students inside a collaborative group. This condition provides 
facilitation of developing skills of the students of the collaborative group through interactions compensating for the 
lack of personal skills. 

• Taking into account individual characteristics of students during the formation of collaborative groups. This condition 
allows removal of limitations of collaboration. 

The students will get an opportunity to develop HOT skills through collaboration owing to: intra-group interactions 
among students if cumulative skills of students in a collaborative group equal the required analytical HOT skills; 
inter-group interactions among students if cumulative skills of students in a collaborative group are less then the 
required analytical HOT skills. 

Example2: 

Guided by the stated conditions of building collaborative group and based on the assessments of analytical skills of 
students (Table 2), two collaborative groups are formed. The first collaborative group g, includes the students Si, s 3 , and 
s 5 . The second collaborative group g 2 includes the students s 2 , s 4 , and s 6 . further development of analytical HOT skills 
of the students in the group g, can be provided by intra-group interactions among students because the group has all 
needed skills. All students of the group g 2 received assessment of the skill “Ordering” equal to zero. It means that the 
students of the group did not develop this skill. Hence, development of this skill in the students of the group g 2 can be 
provided through inter-group interactions with the students from the group g,. 

D. Formation of fixed assessments of HOT skills for the second stage of PBL 

Analytical skills development occurs during the first phase of PBL. Consequently, fixed assessment of this skill type is 
decreased for the second stage of PBL. Furthermore, analysis of assessments of analytical HOT skills reveals a lack or 
inadequate development of certain skills in a study group. It stipulates the need for adaptation of a study group’s 
analytical skills assessments in the process of developing these skills. Adaptation of assessments is realized by changing 
fixed analytical HOT skills assessments, and if examination results reveal a lack of a certain skill development for the 
study group, then fixed assessment of this skill is modified by an instructor. 

Example 3; 

Fixed assessment of the analytical skill type is set equal to 20% for the second stage of PBL. Analysis of the determined 
values of the coefficient of analytical skills development (Example 1) reveals inadequate development of some 
analytical skills in the study group. Owing to that, fixed assessments of “Comparing”, “Selecting”, “Contrasting”, 
“Evaluating”, and “Ordering” are set equal to 2, 3, 4, 5, and 6, accordingly. The fixed assessment is set for the skill 
“Ordering” because this skill has the weakest assessment for the study group after the first phase of PBL. 

An assessment of collaborative skills must be introduced for the stimulation of interactions among students during 
collaborative problem solving. This assessment is set equal to 20%. Fixed assessments of creative skills are set on the 
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basis of preferences of an instructor. Adaptive assessments should provide stimulation and facilitation of acquisition of 
HOT skills and collaborative skills. 

An example of setting fixed assessments of HOT skills for the second stage of PBL is represented by Table 3. 

Table 3. Fixed assessments of HOT skills for second stage of PBL 


The type of thinking skills 

Skills name 

The fixed assessments 


Evaluating 

5 


Selecting 

3 


Contrasting 

4 

Analytical (20%) 

Comparing 

2 


Ordering 

6 


Originality 

15 

Creative (60%) 

Efficiency 

12 


Flexibility 

11 


Problem finding 9 


Elaboration 

13 


E. Assessment of HOT skills development of students after the second stage of PBL 

The instructor composes individual lists of control questions on the basis of the complete list for examination of 
analytical skills development. An individual list can contain some questions for examination of a certain analytical 
skill. The individual questions are formed on the basis of skill assessments of a student after the first stage of PBL. It 
provides adaptive personalized learning. Next, the instructor composes individual lists of control problems based on the 
complete list. Development of the creative skills of a student is assessed by the instructor as a result of solving the 
problems from the individual list. The aggregate of control questions and problems from the individual lists should 
provide examination of all analytical and creative skills. An example of assessment of HOT skills of students after the 
second stage of PBL is represented by Table 4. 

Table 4. Assessments of HOT skills after second stage of PBL 


The 

Skills names 


SI 

S2 

S3 

S4 

S5 

S6 

questions 

q a n 

Comparing 

2% 

1 

2 

1 

2 

2 

1 

q a 2 i 

Selecting 

3% 

3 

2 

3 

2 

1 

3 

q a 3i 

Contrasting 

4% 

4 

3 

3 

4 

3 

4 

q a 4i 

Evaluating 

5% 

5 

4 

4 

5 

3 

4 

q a M 

Ordering 

3% 

2 

2 

3 

3 

3 

3 

q a 52 

Ordering 

3% 

3 

2 

3 

1 

3 

2 


Assessments of analytical skills 

18 

15 

17 

17 

15 

17 

q c n 

Problem finding 

9% 

5 

3 

6 

2 

4 

2 

q c 2i 

Flexibility 

11% 

7 

6 

5 

5 

8 

9 

q c ji 

Efficiency 

12% 

8 

7 

9 

8 

10 

6 

q c 4i 

Elaboration 

13% 

9 

7 

8 

4 

9 

10 

q c 5i 

Originality 

15% 

10 

8 

12 

7 

14 

9 


Assessments of creative skills 

39 

31 

40 

26 

45 

36 


Assessment of HOT skills development is realized by comparison of the values of the coefficient of skill development 
after the second stage of PBL with the corresponding values of the coefficient of skill development after the first 
phase of PBL. The values of the coefficient of developing skills in the study group are determined by formula (1). 

Example 4: 

The values of the coefficient of skill development 5(k ; ) after the first phase of PBL(Example 1) are 
“Comparing” - 0.3; “Selecting” - 0.33; “Contrasting” - 0.36; “Evaluating” - 0.51; and “Ordering” - 0.75 . 

The values of the development coefficient 5(k;) of these skills based on assessments of HOT skills after the second 
phase of PBLtTable 4) are: 

“Comparing” - 0.25; “Selecting” - 0.22; “Contrasting” - 0.13; “Evaluating” - 0.17; and “Ordering” - 0.16. 

Comparison of the specified values of the coefficient of skill development in the study group shows significant 
development of analytical HOT skills due to the second stage of PBL. Particular development of “Evaluating” and 
“Ordering” skills confirms effectiveness of adaptive formation of fixed assessments of HOT skills and collaborative 
PBL. 

Combined assessments of HOT skills of the students are determined as a result of summation of corresponding 
assessments of analytical and creative skills. The combined assessments based on data from Table 4 are presented in 
Table 5. 
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Table 5. Combined assessments of the student HOT skills 



SI 

S2 

S3 

S4 

S5 

S6 

The assessments of analytical skills 

18 

15 

17 

17 

15 

17 

The assessments of creative skills 

39 

31 

40 

26 

45 

36 

The combined assessments of HOT skills 

57 

46 

57 

43 

60 

53 


F. Assessment of collaborative skills of students 

An assessment of collaborative skills of a student is done by taking into account assessments of his (her) HOT skills 
acquired as a result of collaborative PBL. The collaborative group is skill heterogeneous with realization of a condition 
of maximal mutual supplementation of skills of the collaborative group of students. It fosters and facilitates 
skill-sharing among students. 

Assessment of collaborative skills is based on analysis of assessments of HOT skills of students after collaborative 
problem solving. The objective of the analysis is to determine the student (or students) who obtained a maximal 
assessment. Such assessment is a result of the acquisition of HOT skills by students due to collaborative problem 
solving. It can serve as a measure of success of collaboration of a student with other students of the collaborative group 
and also with students of other collaborative groups. Hence, a student with a maximal assessment of HOT skills 
possesses the best collaborative skills. The assessments of collaborative skills of students in a study group are calculated 
proportionally to the assessments of HOT skills on the basis of fixed assessment of collaborative skills. 

The combined summative assessments characterize outcomes of PBL. These assessments are determined by the 
summation of combined assessments of HOT skills and assessments of collaborative skills. 

Example 5: 

A student s 5 has the best collaborative skills since the combined assessment of HOT skills of this student is maximal 
(Table 5). A student s 4 has the worst collaborative skills since the combined assessment of HOT skills of this student is 
minimal (Table 5). 

The fixed assessment of collaborative skills is 20%. This assessment is obtained by the student S 5 since the assessment 
of the student is maximal and equals 60. Hence, the assessment of collaborative skills obtained by the student s 4 equals 
14%. 

Assessments of collaborative skills of remaining students are calculated analogously. The results of calculations are 
shown in the second row of Table 6 . The results of calculating the combined summative assessments of students are 
represented by the last row of Table 6 . 

Table 6 . Assessment of collaborative skills 



SI 

S2 

S3 

S4 

S5 

S6 

The combined assessments of HOT skills 

57 

46 

57 

43 

60 

53 

Assessment of collaborative skills 

19 

15 

19 

14 

20 

17 

The combined summative assessments 

76 

61 

76 

57 

80 

70 


4. Conclusion and Future Work 

This paper has introduced an approach to organizing adaptive PBL leading to the development of HOT skills and 
collaborative skills of students. HOT skills development of students during study of a subject is realized through a 
two-stage process of adaptive PBL. Adaptability of PBL is expressed by changes in fixed instructor assessments caused 
by the dynamics of HOT skills development, dynamic choice of control tests and problems for students, and adaptive 
formation of HOT skills heterogeneous collaborative groups. 

The approach stimulates active, personalized, and collaborative learning, induces the students to develop HOT skills 
and collaborative skills needed for problem solving, and facilitates realizing PBL. Adaptability of PBL is realized by 
taking into account the values of the proposed coefficient of HOT skills development. HOT skills development of 
individual students confirms the effectiveness of adaptive formation of fixed assessments of the skills. Assessments of 
collaborative skills of students are caused by assessments of their HOT skills after collaborative problem solving. 

Further research will be directed towards developing an approach to organizing adaptive PBL, and enhancement of 
efficiency in its practical use. 
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